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CLAIMS 



What is claimed is: 




V+^AVnethod for providing collateral information for inclusion with an 
information stream, comprising steps of: 

examining\he information stream to recognize a presence of events that occur 
in the information stream; 

automatically generating database queries from recognized events; and 

analyzing database query results so as to rank and select database query results 
to be inserted int6 the information stream as collateral information. 

2. A method as in claim 1, wherein the step of analyzing comprises a step of 
ranking the database query results based on a plurality of criteria. 

3. A method as in claim 2, wherein the plurality of criteria comprise a score 
derived from a free text searchW the database using text that is automatically extracted 
from the information stream, on a number of named entities appearing in the text and 
in the database query results, ana on a taxonomy path score, where the taxonomy path 
score represents an amount of rfclatedness between a taxonomy-related information 
element found in the text and a predetermined taxonomy tree. 

4. A method as in claim lAwherein the database queries are automatically 
generated based on information corresponding to a list that identifies topics in text that 
is automatically extracted from the information stream, where the topics correspond to 
predetermined topic taxonomies. \ 

5. A method as in claim 1, wherein the step of examining comprises a step of 
automatically extracting text from the information stream. 



6. A method as in claim 5, wherein the srep of examining further comprises steps 
of: \ 



segmenting the text into sentences; and 



4 
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operating on thejsentences to identify topics that correspond to predetermined 
topic taxonomies, wherein the step of automatically generating database queries 
operates on identified topics. 

7. A method as in claim 5, wherein the step of automatically extracting text from 
the information stream comprises a step of operating a voice recognition system. 

8. A method as tn claim 5, wherein the step of automatically extracting text from 
the information stream comprises a step of extracting closed caption text. 

9. A method as in claim 5, wherein the step of automatically extracting text from 
the information stream! comprises a step of operating a character recognition system. 

10. A method afe in claim 5, wherein the step of automatically extracting text 
from the information stneam comprises a step of also generating text that is descriptive 
of a number of human ff ces that are present in an image conveyed by the information 
stream. 



1 1. A method as in claim 5, wherein the step of examining the information 
stream further comprises Weps of segmenting the text into sentences and a step of 
operating on the sentences to identify topics that correspond to predetermined topic 
taxonomies and the preserve of names of entities, and further comprising steps of 
assembling a list comprisedpf an identified topic having a start time and an end time, 
as well as any named entities that occur between the start time and the end time, 
assembling a query object comprised of named entities that occur between the start time 
and the end time of the identified topic, searching at least one database to identify a first 
set of stored documents that correspond to the topic, identifying a subset of the first set 
of documents that contain the named entities, identifying a second set of documents that 
correspond to words found in the text; scoring the returned documents based on a 
plurality of criteria and ranking the documents based on their scores. 



fQrr A method for prolvid 
information stream, comprisir 



ling collateral information for multiplexing with an 
g steps of: 



converting the information stream into text; 
analyzing the text to identify information elements; 
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automatically generating queries from the information elements for searching at 
least oneVlatabase; 

extracting tyata from database search results that is relevant to the information 
stream; anc 

multiplexing the data into the information stream for presentation at a 
destination or the information stream. 

13. A method as in claim 12, wherein the step of extracting comprises a step of 
ranking the extracted mformation based on a plurality of criteria, and where the step of 
multiplexing uses the ranked data. 

14. A method asm claim 12, wherein the step of extracting comprises a step of 
ranking extracted docum&nt information based on a score derived from a free text search 
of a document database using the text, on a number of named entities extracted from the 
text that are found in the documents, and on a taxonomy path score, where the taxonomy 
path score represents an amount of relatedness between a taxonomy-related information 
element identified in the tex\ and a predetermined taxonomy tree. 

15. A method as in claim 12, wherein the queries are generated based on 
information elements that correspond to a list of information elements identifying topics 
in the text being analyzed, wP^ere the topics correspond to predetermined topic 
taxonomies. 

16. A method as in claim 12,Yvherein the step of analyzing the text comprises 
steps of segmenting the text into sentences and a step of operating on the sentences to 
identify topics that correspond to predetermined topic taxonomies, and wherein the step 
of automatically generating queries operates on identified topics. 

17. A method as in claim 12, wherem the step of analyzing the text comprises 
steps of at least segmenting the text into sentences, identifying names of entities within 
the text, and a step of operating on the sentences to identify topics that correspond to 
predetermined topic taxonomies, and wherein the step of automatically generating 
queries operates on identified topics and ranks the database search results based at least 
on numbers of named entities found and on an amount of relatedness between a 
taxonomy-related information element identified in the text and a predetermined 
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taxonomy tree. 

A system for providing collateral information for inclusion with an 
information stream, said system operating in real time or substantially real time and 
comprisingK 

a subsystem for examining the information stream to recognize a presence of 
events mat occur in the information stream; 

a subsystem, having an input coupled to an output of said examination 
subsystem, Yor automatically generating database queries from recognized 
events; \ 

a database for receiving said database queries; and 

a subsystem, having an input coupled to an output of said database, for analyzing 
database query results so as to rank and select database query results to be 
inserted into the information stream as collateral information. 

19. A system as in claim 18, wherein the analyzing subsystem employs ranking 
criteria comprised of a score delved from a free text search of the database using text 
that is automatically extracted from the information stream, on a number of named 
entities appearing in the text and in\he database query results, and on a taxonomy path 
score, where the taxonomy path score represents an amount of relatedness between a 
taxonomy-related information elementYound in the text and a predetermined taxonomy 
tree, and wherein the query generation subsystem generates queries based on 
information corresponding to a list that identifies topics in the text that is automatically 
extracted from the information stream, whete the topics correspond to elements of the 
taxonomy tree. \ 

20. A system as in claim 18, wherein said examining subsystem comprises at 
least one unit for automatically extracting text from the information stream, a unit for 
segmenting the text into sentences and at least oneVinit for operating on the sentences 
to identify topics that correspond to predetermined topic taxonomies, wherein said query 
generation subsystem automatically generates databasV queries based at least in part on 
identified topics. \ 
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2 1 . A system as in claim 20, wherein said text extracting unit comprises at least 
one of a voice recognition system, a system for extracting closed caption text, and a 
character recognition system. 

22. A system as in claim 20, wherein said examining subsystem comprises a unit 
for generating textVhat is descriptive of a number of human faces that are present in an 
image conveyed byihe information stream. 

23. A system a\ in claim 20, and further comprising a unit for operating on the 
sentences to identify theWesence of names of entities, and further comprising a unit for 
assembling a list comprised of an identified topic having a start time and an end time, 
as well as any named entities that occur between the start time and the end time, and 
where the query generation^ subsystem assembles a query object comprised of named 
entities that occur between me start time and the end time of the identified topic for 
searching said database to identify a first set of stored documents that correspond to the 
topic, a subset of the first set oftdocuments that contain the named entities, a second set 
of documents that correspond to words found in the text; and where said analyzing 
subsystem scores the returned documents based on a plurality of criteria and ranks the 
documents based on their scores) 

J)X^ fi^rA computer-implemented method for generating collateral information from 
an audio/video stream, comprising Steps of: 



x: 



examining the audio/video stream to recognize a presence of events that occur 
in the audio/video stream; 

generating database queries fromXrecognized events; and 



analyzing database query results so is to rank and select database query results 
to be presented as the collateral information. 

\ JZ&f'p^ method as in claim J^^ifHur^er comprising a step of inserting the 
collateral information into the audio/yi^eo streair^n real time or substantially real time. 

Qjcy 27T A method as in cimm^, wherein the srep of examining includes a step of 
generating a speech transcript from at least the audio portion of the audio/visual stream, 
and wherein recognised events comprise speech topics 
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2-7 28^ A method as in claim 
television broadcast signal. 



wherein the/audio/visual stream originates as a 



2$ 2?f. A method as in claim 2^ wherein ihe audio/visual stream originates at a 
meeting, and further comprising a step of presenting the collateral information to 
meeting participants in real time or substantially real time. 

\ / 
2J\ A method as in claimj^, wherein the step of presenting comprises a step of 

inserting me collateral information into tne audio/video stream, and displaying the 

audio/video stream to the meeting participants. 

\ .2? / 

fyQ . A method as in claim ^9, ancyfurther comprising a step of archiving at least 
the collateral iniWmation. 

\ 

3f 32TAmeth\d as in claim j^/wherein the database queries are automatically 
generated based on information corresponding to identified topics extracted from the 
audio/video stream, wnere the topids correspond to predetermined topic taxonomies. 

\ T A 

method as m claim J^o, wherein the step of examining includes steps of 
generating a speech transcript comprised of words from at least the audio portion of the 
audio/visual stream; segmrating the words into sentences; and operating on the 
sentences to identify topics thaWorrespond to predetermined topic taxonomies, wherein 
the step of generating database queries operates on identified topics. 

J>3 0^ A computer readable Vnedia having recorded thereon a program for 
providing collateral information for inclusion with an information stream, the program 
comprising instructions for examinin^he information stream to recognize a presence 
of events that occur in the information Vtream, for automatically generating database 
queries from recognized events; and for analyzing database query results so as to rank 
and select database query results to be inserted into the information stream as collateral 
information. \ 



